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Abstract 



Public-key cryptosystems are suggested based on invariants of groups. We give 
also an overview of known cryptosystems which involve groups. 

1 Introduction 

In public-key cryptography the problem is to produce a cryptosystem which contains the 
following ingredients: a public key k e , a secret key k d , a public encrypting function f e 
and a secret decrypting function fd. If somebody (usually named Bob) wants to send a 
message m to another person (usually named Alice) via a public channel then he transmits 
an encryption u = f e (m, k e ). To decrypt a message Alice calculates m = fd(u, kd). It is 
supposed that kd, fd are known only to Alice, while k e , f e are known publically. Another 
important property of a cryptosystem is that an unauthorized person (named Charlie) 
would be unable to learn m from u (without knowing k d , fa). 

A lot of efforts were undertaken to design cryptosystems (some literature one can find 
in Jl7|, [13|, [11]]). Still for no cryptosystem its security is proved and the issue of security 
remains a challenging problem. All the existing results on security concern impossibility 
of breaking a cryptosystem by certain fixed means, say, in frames of a particular proof 
system. But what appeared to be interesting in the course of developpment of cryptogra- 
phy is that many connections with other areas of mathematics were discovered. In fact, 
public-key cryptography in many aspects plays a role of a bridge between mathematics and 
computer science. The most recognized cryptosystems are based on number-theoretical 



ideas like RSA, Diffie-Hellman or the elliptic curves cryptosystems (see e.g., |T7|], in this 
book one can find also some cryptosystems invoking combinatorial-algebraic NP-hard 
problems) . 

In these notes we study cryptosystems which involve ideas from the theory of group 
invariants. Several known cryptosystems rely on groups (below we give a short overview 
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of them), but surprisingly the concept of a group representation invariant was never 
exploited, although it fits quite well the aims of cryptography. Still, applying invariants 
into cryptography encounters the similar difficulties as other known approaches, however, 
the purpose of these notes is to introduce invariants into cryptography, design a possible 
cryptosystem and to discuss arising problems on its security 

Thus, the general idea behind using groups (and their invariants) is as follows. Let 
E denote the set of encrypted messages and a group G act G : E — > E. Some examples 
of E are the set of words in a certain alphabet or a vector space. In addition, a subset 
M C E of plaintext messages is distinguished being transversal to the orbits of G, i.e. no 
two distinct messages m 1 ,m 2 G M belong to the same orbit of G. 



Then G is used for a probabilistic encryption |T^] being more efficient in "hiding 
information" than a deterministic one. Thus, for encrypting a message m G M Bob picks 
randomly an element g from the group G and transmits gm G E via the public channel. 
Alice has to decrypt gm to learn m applying her secret key. And Charlie has to be unable 
to learn m from gm. 

Usually, the latter two properties of the cryptosystem are achieved by a special choice 



of G. In the widely used quadratic residue cryptosystem |TJ|, [13] one takes an integer 
n = pq for large primes p, q (being a secret key of Alice). As the group G one takes (Z*) 2 
and E = {g G Z* : J n (g) = 1} where J n denotes the Jacobi symbol. Take also an element 
a G E — G (being a non-square), put M — {1, a}. A public key consists of n and a. Thus, 
to encrypt 1 Bob picks first a random g G Z*, then its square g 2 G G, and to encrypt a 
Bob picks g 2 a being a random non-square in E, clearly E = GU Ga. 

The task of Alice is to verify whether an element b G E (being a transmitted encrypted 
message) is a square. This Alice can easily do using p, q and Legendre- Jacobi symbols 
J p , J q . On the other hand, it is a common belief that Charlie is unable to verify whether 
b is a square without knowing p, q. 

The described quadratic residue cryptosystem was generalized to a class of cryptosys- 
tems called homomorphic. Namely, let / : E — > H be a group epimorphism which is a 
secret key of Alice. There is an exact sequence of group homomorphisms 

B^E^H^O 

and a public key is B, s, E, H and a subset M C E transversal to G — ker(/), hence / 
provides a bijection between M and H. This is consistent with the above notations: G 
acts (by the multiplication from the left) on E and the set of plaintext messages M being 
transversal to this action, but here G is given implicitly as an image s(B). To encrypt a 
message m G M Bob picks randomly b G B and transmits s(b)m. Alice decrypts s(b)m 
applying /, taking into account that f(s(b)m) = f(m). For Charlie it is difficult to 
decrypt without knowing /. In the described above quadratic residue cryptosystem we 
have H = Z 2 , B = E, s(b) = b 2 and / being the epimorphism of the quadratic residue. 
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In ||, [JJJJ a question was posed for which groups H homomorphic systems can be con- 
structed (more generally, one could consider rings rather than groups homomorphisms)? 
For some abelian groups H cryptosystems were designed in j|, [23], ^ |25|, pTfl . For certain 



diedral groups H cryptosystems are designed in [28]. In [|Lq] a homomorphic system was 



designed for any solvable group H. For cryptosystems over elliptic curves see |T^, 17 ]. 

What is common in all the mentioned constructions is that decrypting relies on the 
knowledge of secret primes p, q. In these notes we suggest another way of decrypting 
(and encrypting) based on an invariant w : E — > F, i.e. w being constant on the orbits 
of G. Then Alice is able to decrypt an encrypted message gm by means of calculating 
w(gm) = w(m), provided that w takes distinct values on the elements (plaintext messages) 
from M. 



The theory of invariants, see e.g. [[7J, [JO], [Jj]], is developped mostly in the situation 
when G : E — > E is a linear representation, so E is a vector space and G C GL(E) over 
a field F and w being a polynomial. But perhaps, it would be also worthwhile to look at 
other group actions and their invariants. 

Since for not too many infinite series of linear representations G C GL(E) their 



invariants w are known explicitly and can be calculated fast ([0), we suggest to hide G 
considering its conjugation a~ x Ga C GL(E) for a secret matrix a G GL(E). Then an 
invariant e — > w(ae) of the conjugation a~ x Ga enables Alice to decrypt a message a~ 1 gam 
where a~ l ga is a random element from a~ x Ga chosen by Bob for encrypting a message m. 
Usually, the group a _1 Ga (being a public key together with E and M C E) is given by a 
set of matrices in GL(E) being its generators. It is a quite succinct way of representing a 
group by a set of its generators, in particular, known finite simple groups are representable 
just by two generators, and any finite group G is representable by log 2 |G| generators. In 
calculations with G represented by a set of generators it is not necessary to assume that G 
is finite (which is the case in particular, when the field F is finite) because for encrypting 
Bob has just to pick randomly a certain product of generators of a _1 Ga. 

In the next sections we describe cryptosystems based on group invariants and dis- 
cuss the issues of their security, but first we complete an overview by two families of 
cryptographic tools which involve groups. 

Another particular problem of cryptography, apart from designing cryptosystems and 
closely connected with it, is the key agreement protocol, see e.g. |TTJ, [T7| . Now Alice 
and Bob want to agree about a common key communicating via a public channel. The 
usual approach is to choose by each of them secretly commutating operators Ja (by 
Alice) and (by Bob) in the same set E and in addition a certain (public) e e E. Then 
Alice communicates Bob communicates /s(e) and they agree on a common key 

/a(/b(c)) = /b(/a(c)). In the first key agreement protocol due to Diffie-Hellman (see e.g. 
|T3| , |l7j) it was used /^(e) = e a ,/s(e) = e 6 (modp) for integers a, b. Thus, decrypting (by 
Charlie) of Diffie-Hellman protocol relates to computing the discrete logarithm which is 
believed to be difficult, its complexity was studied in fl6], [22" . 
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This general approach was considered in the following setting (see || |OJ . Let E 
be a group with two subgroups Ea, Eb C E which commute with each other. Then as 
fA Alice chooses a conjugation e — > a~ x ea for a randomly picked a e E 1 ^, respectively, 
/s( e ) = & _1 e6 for 6 6 i?B. In jrjj the braid group is used as and the difficulty of 
breaking this key agreement protocol relates to the difficulty of the conjugacy problem in 
the braid group. 

Few cryptosystems based on the difficulty of the word problem in appropriate groups 
were proposed in 0, || [TO], 



The last family of cryptosystems we mention rely on lattices (being discrete abelian 
subgroups of M. n ), the first such a construction is due to Let L C M n be an n- 
dimensional lattice with a property that it contains a (hidden) (n — l)-dimensional sub- 
lattice V whose linear span being a hyperplane H, satisfying the following property. The 
coset hyperplanes Hi parallel to H such that Ujifj D L are well separated: the distance 
between any adjacent pair Hi, Hi + \ of them is greater than a suitable large d. In other 
terms, there is a basis of L which consists of n — 1 rather "short" vectors which form a 
set C from V and a single "long" vector c having a "big" coordinate orthogonal to H. 

Then a cryptosystem from |I] considers a random basis C\ of L as a public key and a 
basis C U {c} as a secret key. The plaintext is encrypted by Bob by a random vector 
from L, and the plaintext 1 is encrypted by a random vector from W 1 . 

For decrypting a vector u Alice computes a magnitude I = (u,c — P#(c))/| \c — Ph{c) 1 1 2 
where P# denotes the orthogonal projector onto the hyperplane H. If I is an integer 
(this means that u lies on a certain coset hyperplane Hi) then Alice can declare that the 
plaintext message of Bob is (otherwise, 1). Actually, in this manner Alice recognizes 
elements from UiHi rather than from L, so an error happens when u G Uj-Hj — L. To 
correct this Bob slightly perturbs u, so each point from L is surrounded by a ball of a 
suitable radius r in order to cover Uj-ffj, but on the other hand, not to cover the whole 
space M n . Moreover, the perturbations of adjacent hyperplanes H { and H i+1 should be well 
separated, just for this reason the condition on a "long" vector c was imposed. Finally, 
Alice decrypts the points at the distance at most r from the union UiHi as 0, otherwise 
as 1. Still, an error could happen when u lies at the distance at most r from the union 
Ujf/j, but now it is more probable that Bob encrypted in this case (rather than 1). 

Thus, the presumed difficulty of breaking (by Charlie) this cryptosystem relies on 
finding a long vector in a lattice given by its basis C\ (or equivalently, a short one in the 
dual lattice), provided that a long vector is unique in an appropriate strong sense. 

Another cryptosystem based on perturbations of a lattice was designed in Jl2| . Here a 
plaintext message is a point of a lattice L C W 1 and its encryption is its small perturbation 
in W 1 . Then the problem of breaking the cryptosystem leads to finding for a given real 
point the closest to it vector in the lattice L. This problem is known to be NP-hard, as 
well as its approximating up to a constant factor. To make the decrypting possible Alice 
first chooses (randomly) a basis C\, . . . , c n of L of a special form (namely, such that the 
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magnitude Yli Il c i||/| det(cj)| is not too large, a basis with this property is called "almost 
rectangular") which serves as a secret key. After that the basis C\, . . . , c n is (randomly) 
spoiled and the resulting new basis of L serves as a public key The point is that an almost 
rectangular basis allows Alice to find the closest vector in L, provided that a perturbation 
was small enough. 

Thus, in both mentioned lattice-based methods [1], 12] a plaintext message is hidden 
by a small perturbation, this differs from our suggestion to hide by means of shifting by 
an element from a certain group. 



2 Construction of cryptosystems based on group in- 
variants 

Let G C GL n (F) be a representation of a group G where one can deem w.l.o.g. a field 
F to be algebraically closed, however in computations the entries of the matrices from G 
could belong to a certain subfield of F, it is reasonable, for example, the entries to belong 
to a finite subfield. 

Assume that we know a (non-constant) invariant w ([[?|, |3(|) of the representation 
of G, i.e. a polynomial w G F[Xi, . . . ,X n ] such that for any element g G G and any 
vector v G F n we have w(gv) = w(v). Besides, we fix a pair of nonzero distinct vectors 
v ,v 1 eF n . 

Usually (and we suppose this) one is able to generate elements from G. To design a 
(probabilistic) public-key cryptosystem Alice chooses randomly a matrix a G GL n (F) with 
the property that w(avo) ^ w{av\) (clearly, almost any matrix a satisfies this property). 

Public key: fo,fi and a set of elements of the form hi = a~ l gia G GL n (F) where g^ 
being randomly generated elements of G. 

Secret key: a 

Encryption: a letter or respectively, 1 of a plaintext message is transmitted as a vector 
u = hi x • ■ • h^VQ (or respectively, u = ■ ■ ■ h it vi) for randomly chosen z 1; . . . , %\. 

Decryption: given a vector u G F n Alice computes w(au) and verifies whether it equals 
to w(av ) (in this case the plaintext message was v since w(av ) = w^ah^ ■ ■ ■ h^vo) = 
w(au)) or to w(av\) (in this case the plaintext message was V\). 



3 Discussion on the security 

To break the designed cryptosystem Charlie can try to find a certain invariant w' of 
a (sub)group H of a conjugation a~ l Ga (where H is given by a set of generators hi). 
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One can think of the group G C GL n (F) (of exponential in n size or even infinite) 
to be known as well as an invariant w. Charlie could try to look for an invariant w' 
in the form w'(v) = w(bv) for an unknown matrix b G GL n (F). Let d denote the 
degree of w (as usually, in the invariant theory one might reduce consideration to a 
homogeneous w). Substituting w' into the known generators of the form a~ 1 g i a of H, so 
w(bv) = wib^a^giC^v) for each i (clearly, any such matrix b would fit), and equating the 
coefficients at all the monomials in n coordinates of a vector v, Charlie obtains a system 
of polynomial equations in the entries of a matrix b of degrees d. Hence this polynomial 
system contains ( n+ d _1 ) equations (for each i) of degrees d in n 2 variables being the entries 
of b. 

Alternatively, Charlie could search for an invariant w' treating it as a polynomial 
of degree d with indeterminate ( n+ ^ _1 ) coefficients satisfying equations w'(hiv) = w'(v) 
for each i for any vector v G F n . This provides a linear system in the indeterminate 
coefficients. 

Anyway, the complexity of both procedures depends on ( n+ j -1 ), therefore, for the 
security reasons one should take a group G without invariants of degrees d less than 
const • n. On the other hand, the invariant w should be computable within polynomial in 
n complexity (below we give few such examples). 

The problem of finding an invariant seems to be difficult and in general not much is 
known beyond obvious applying the Reynolds averaging operator | C | 1 J2 9( zg9 (provided 
that G is finite), cf. pH[ . 

Let us consider two other approaches towards breaking the described cryptosystem. 

In the first approach Charlie tries to find a matrix a (or any other b G GL n (F) such 
that bHb^ 1 C G). Clearly, one can assume w.l.o.g. that H is conjugate to G itself 
(rather than to a certain its subgroup) taking clS Qi db set of generators of G (randomly 
chosen large enough set of ^ generates G with a high probability). Then testing an 
existence of b (and finding if it does exist) such that bHb^ 1 = G is called the conjugacy 
problem for matrix groups. One can reduce to the latter problem the conjugacy problem 
for permutation groups as it was communicated to the author by Eugene Luks [21] . In 
its turn, the difficulty of the conjugacy problem for permutation groups was conjectured 
in |2(J where its complexity was posed as an open question. Furthermore, the graph 
isomorphism problem is reducible to the conjugacy problem for permutation groups . 
Thus, the first approach by Charlie leads in particular, to the graph isomorphism problem. 

In the second approach Charlie tries to find a matrix h from H such that hu = vq 
(or respectively, hu = i>i). This problem (in a particular case when if is a permutation 
group) is called the vector transporter problem |2(J where its difficulty was conjectured. 
Again the graph isomorphism problem is reducible to the vector transporter problem |20| . 

We observe that a particular case of the vector transporter problem when (the set of 
encrypted messages) F 4 is the space of 2 x 2 matrices and a group H = SX 2 (Z) x SL 2 { r l>) 
acts on F A by v — > h\vh 2) where (hi, h 2 ) G H, was proved to be NP-hard for the average 
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complexity In this connection we mention that a hardness of breaking a cryptosystem 
for the average complexity would be more desirable than a hardness for the worst-case 
complexity (cf. fT9|j ). 

Thus, both approaches towards breaking the described above cryptosystem relate to 
the graph isomorphism problem. But of course, the difficulty of proving a reduction 
of the graph isomorphism to breaking the cryptosystem lies particularly, in finding an 
appropriate invariant w for a group G such that the graph isomorphism problem can be 
reduced to the conjugacy and to the vector transporter problems for G. It would be 
interesting to understand whether the graph isomorphism problem indeed, is reducible to 
breaking the described above cryptosystem. 

We mention also that a similarly looking problem of equivalence of representations of 
the group algebras F[H] (rather than the groups), in other words finding a matrix 

a G GL n (F) such that a~ 1 F[H]a = F[G], can be solved over an algebraically closed field 
F PP[ taking into account the structural theorems of Schur and Wedderburn. 



An evident remark is that to provide more security of the cryptosystem it would be 
reasonable to change secret and public keys a and a^g^a quite often. 

Let us give few simple examples of cryptosystems based on invariants of classical 
groups J?], |30 . 



Example 1 (|30|). As a group G we take a subgroup of GL n generated by the 



symmetric group S n permuting the standard basis e^, 1 < i < n and all the matrices t 
such that tei = CiCi where C™ — 1, 1 < i < n, (ci • ■ •c n ) / = 1 for some l\m. Then as w 
one can take the power sum x™ + . . . + x™. We deliberately consider an extension of the 
symmetric group to avoid invariants of small degrees (see the beginning of this section). 

Example 2. Consider the representation of the group G = SL n (F) on the symmetric 
product S 2 F n , in other words, on symmetric matrices (or quadratic forms) by v — > mvm T 
where m G G and T denotes the transposing. Then as w one takes det(u). 

Example 3. Now G = GL n (F) which acts on the direct sum F 2 ™ 2 = F n © • • • © F n of 
2n copies of F n by m(pi, . . . ,P2n) = ( m Pi, ■ ■ ■ , m P2n)- Consider two (disjoint) partitions 
I\ U Ji = I 2 U J 2 = {1, . . . , 2n} into n-element subsets = \J\\ = \I 2 \ = \J 2 \ = n. As 
det^ we denote the determinant of n vectors Pi for % G I\. Then as w we take the rational 
invariant 

det^ det jj 
det/ 2 det j 2 

In the described above cryptosystem we considered polynomial invariants w, but nothing 
changes when we deal with rational invariants w G F(Xi, . . . , X n ), except that, of course, 
in the cryptosystem Alice should pick vectors Vq,Vi G F 2n and a matrix a G GL 2n 2(F) 
in such a way that w(av ), w(av i) be defined. 
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One could produce more similar examples invoking direct, tensor, symmetric, exterior 
products of group representations. 

The present state of art of cryptography does not allow to prove security of cryp- 
tosystems, the latter is usually a question of belief in a difficulty of a relevant problem 
and a matter of experience (that is why it is not quite unusual to have a paper on cryp- 
tography without theorems, including this one). Just the opposite, one could expect a 
"disappointing" breaking of a particular cryptosystem. This is not excluded for any of the 
aforementioned examples (and avoiding solving the graph isomorphism problem, see the 
discussion above). On the other hand, such breaking would lead perhaps, to interesting 
algorithms in group representations. Thus, one can treat the examples (and the general 
construction in all) just as a suggestion to play with cryptosystems based on the invariant 
theory. 
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